System and Methods For Computerized Safety and Security

ABSTRACT

Systems and methods are provided for measuring, assessing, predicting, improving and presenting the state of physical object temperatures using imaging devices, e.g., a thermal infrared camera, and/or intruders in a region of interest to an operator, such that little or no operator effort is required to install, use or receive reports from the system. The invention also includes, for example, means and methods for exploiting autonomous operation and configuration, placement at remote sites, enhancement of image resolution and estimation of range such that accuracy of results and autonomy of operation is enhanced.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/800,475, filed Feb. 2, 2019, the entire contents of which arehereby incorporated by reference.

TECHNICAL FIELD

The present invention generally relates to sensor data collection andprocessing for safety and security. More particularly, the presentinvention relates to thermal and color image sensor data collection andimage processing for the purpose of industrial site security, e.g.,perimeter security and safety, via thermographic measurement ofindustrial assets in space and time.

BACKGROUND

In the field of industrial thermography, the current standard practiceis to use handheld devices to make image measurements and subsequentlycombine these with additional supporting physical measurements (e.g.,atmospheric conditions for absorption). Sequences of manual operationsare often combined with computer assisted operations to produce reportscorresponding to the point in time at which the handheld measurementswere made.

Thus, present day practices for the measurement of the physicaltemperature of industrial equipment often involve human measurement,e.g., with a handheld thermographic device, followed by manualmeasurements and assessments of contributing factors. Contributingfactors may include, for example, equipment optical properties,environmental properties, and sources of thermal energy other than theequipment being assessed. Such additional assessments are made toincrease the accuracy of the equipment temperature reported by thehandheld thermographic device.

Such manual measurements can be valuable to the owners and operators ofequipment, but often the equipment being assessed is in a dangerousarea, e.g., high voltage electrical transformers, or in a dangerousstate, e.g., on the verge of exploding due to transformer oil nearingits flash point. Further, since underlying thermal processes for themeasured equipment typically vary on a scale of minutes or hours, makinga single measurement on a yearly or even a monthly scale can lead toerroneous indicators of health and status.

At the same time, there is also a known risk of malevolent humanintrusion at some equipment sites, either for the purposes of theft orsabotage, and these also endanger both industrial assets and the humanswho visit them in order to assess physical temperature or make otherassessments of physical condition that affect performance and utility ofthe equipment. Consequently, it is advantageous to use both security andthermography functions so as to minimize injury to equipment or humanswho use or visit the equipment.

The present invention addresses these and other limitations of the priorart.

SUMMARY OF THE INVENTION

The following is a summary of the invention intended to provide a basicunderstanding of some aspects of the invention. This summary is notintended to identify key or critical elements of the invention or todelineate the scope of the invention. Its sole purpose is to presentvarious concepts of the invention in a simplified form as a prelude tothe more detailed description and the defining claims that are presentedlater.

The present invention relates to systems and methods for measuring,assessing, predicting, improving, and presenting the state of physicalobject temperatures using imaging devices, e.g., a thermal infraredcamera, and/or intruders in a region of interest to an operator, suchthat little or no operator effort is required to install, use, orreceive reports from the system.

These and other features and advantages of the invention will beapparent to those skilled in the art from the following detaileddescription of preferred embodiments, taken together with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1, is a block diagram of an embodiment of the invention;

FIG. 2, is a block diagram illustrating an embodiment of automaticallyproducing an object catalogue for a site;

FIG. 3, is a block diagram illustrating an embodiment of estimatingobject range using the system of FIG. 1

FIG. 4 is a block diagram illustrating an embodiment of super-resolvingan image using a gimbal;

FIG. 5 is a block diagram illustrating an embodiment of using theinvention for automated thermography and security;

FIG. 6 is a block diagram of an embodiment of the platform used inremote installations of the invention;

FIG. 7 is an exemplary thermal image useful in describing variousaspects of the present invention; and

FIG. 8 is a graph of absolute temperature difference associated with aportion of the objects illustrated in FIG. 7.

DETAILED DESCRIPTION OF PREFERRED EXEMPLARY EMBODIMENTS

In general, the present invention relates to the automation ofindustrial thermography. In that regard, the following detaileddescription is merely exemplary in nature and is not intended to limitthe inventions or the application and uses of the inventions describedherein. Furthermore, there is no intention to be bound by any theorypresented in the preceding background or the following detaileddescription. In the interest of brevity, conventional techniques andcomponents related to thermal imaging, image processing, computerprocessors, robotics, and calibration methods may not be described indetail herein as such topics are well known by those of ordinary skillin the art.

In accordance with one embodiment, in order to avoid unnecessary humanrisk resulting from measurement and/or unwanted intrusion or sabotage,the present invention enables the automation of thermographicmeasurement and intrusion detection such that a single system mitigatesrisk of harm to equipment and humans in an enterprise. Toward that end,embodiments of the present invention relate to an autonomous industrialsecurity and safety system including one or more imaging devices withintegral computing and data storage capacities configured in a networkto which additional computers and storage devices may be connected, andto which a user may connect in order to access raw and processed data,and from which a user may receive automated communications concerningthe current and likely future state of the physical assets beingmonitored. In accordance with one embodiment, the imaging device(s)comprise a multispectral imaging system having multiple axes of motionsuch that the fields of regard for the imaging devices can be changedthrough actuation, e.g., gimbal motion, in one or more axes, e.g., apan-tilt gimbal, so as to produce a system field of view larger thanthat of a single image device. In this regard, the term “gimbalassembly” or the like is used herein without loss of generality. Anyform of robotic system or multi-axis linkage system may be used toeffect motion of the thermal infrared cameras.

In accordance with various embodiments, at least one non-visual (e.g.,thermal infrared) image device is used for imaging objects in its systemfield of view, so that the thermal emission of imaged objects can beused to estimate object physical temperature from the measured thermalinfrared radiance imaged by the thermal camera. Security function, e.g.,perimeter security, is enabled by using thermal or other sensor orcamera data to detect and report the presence of human, human-like, orhuman-related activity in areas for which such activity is of concern,e.g., prohibited to owners or operators of the industrial site.Computers that are integral to system imaging devices, e.g., onsite orembedded computers, may produce some or all of the data productsrequired to achieve the security and safety functions.

Computing devices (e.g., desktop computers, laptop computers, tabletcomputers, smartphones, or the like) that are connected to systemimaging devices by way of a network connection may also produce dataproducts and will often be used for both the production of data productsand associated reports, graphs, alerts and other results of interest toowners or operators of the industrial equipment being monitored.Security function extends to using cueing devices located remote fromthe invention that detect events and transmit signals received by theinvention that it interprets and uses in a “slew-to-cue” fashion, usinga gimbal to position the field of view, e.g., for the thermal infraredcamera, in proximity to the device transmitting the signals.

The invention can be configured for equipment and intrusion monitoringmanually by an operator, e.g., sitting near the invention or remotely ata desk over a network, or for automatic configuration. Automaticconfiguration of the system may involve, for example: using one or morecomputers to control the invention so as to survey its surroundings,detect and identify relevant objects or spaces, classify those objectsand spaces, estimate their physical properties and the physicalproperties of their environment (e.g., such as would contribute to andaffect an interpretation of temperature based on measurements, includingbut not limited to thermal radiance of nearby objects, atmospheric lossand scattering as a function of path length, optical path length, etc.),compute the movements needed to capture their data, schedule the datacapture based on predetermined or statistically estimated risks andphysical behavior (e.g., maximum rate of change) and initialize datacollection databases in local storage and remote storage, e.g., “cloud”storage.

Automation enabled by embodiments of the invention also includes the useof gimbal and imaging devices to localize equipment or spaces, e.g.,estimate distances from imaging devices to equipment or spaces in themeasurement area, and combine it with GPS or other locationingtechniques to determine its position on a map, and the production ofenhanced resolution (or “super-resolution) using predetermined ormeasured properties of the imaging devices and related optics (e.g.,point spread function). The invention also addresses the automation ofthe estimation of calibration parameters and equipment physicalparameters, e.g., emissivity, using physical observables in theenvironment, e.g., atmosphere/sky, stellar objects, identifiable solids,and historical data on such objects and materials, such historical dataproviding time varying observables against which one may estimate, e.g.,via regression or otherwise statistical methods, unknowns that thenpermit accurate assessment of observables. The present inventioncontemplates achieving such automation through the use of data alone orin combination with physical and mathematical models of underlyingphenomena.

Autonomous Operation at Site

In accordance with one embodiment of the invention, a site may berapidly and autonomously monitored for both thermography and security.With reference to the conceptual block diagram of FIG. 1 in conjunctionwith the flow chart of FIG. 5, an exemplary flow 500 of an embodiment ofthe invention will now be described. As shown, the method begins (atstep 501) with the placing of a system (FIG. 1) at a known geographiclocation. In accordance with one embodiment, this placement is madestraightforward by virtue of the use of a mobile platform 106 andenclosure 108 having autonomous means of supplying its power, e.g.,solar or wind or equivalent autonomous (or self-contained) power source110, so that placement constitutes an installation once a user enablespower, e.g., with a power switch, for the system (step 502).

The system then finds a network (step 503), e.g., wireless LTE or WiFimesh, or physically connected network should such be available at thesite, and connects (via network interfaces 109, 114, and 117) (step 504)to a remote server 115 preconfigured for use with the system.Subsequently, the system will send GPS, e.g., from a GPS auxiliarysensor or from a GPS integrated into the LTE radio etc., and a uniquesystem identifier (step 505) preconfigured at the time of systemmanufacture to the remote server 115. The system then is able to receiveits initial tour and security settings (step 506). The “tour” describesa sequence of locations at which the system collects data and transmitsthe data to the remote server 115 such that thermographic data can becollected for objects imaged at each location. The “security settings”describe the spaces for which thermal and/or color video are streamed tothe local computer 101 and also the remote server 115 such and datastorage 116 hat a user might review prior video data or view live videodata on a remote computer 118 on its computer monitor 119 or equivalentdisplay device. Data may also be stored on local data storage 102.

Given an attitude, heading and reference system (AHRS) device connectedto an auxiliary sensor input 111 the AHRS can be used to orient thegimbal 112 (step 507) such that the initial tour and security settingshave relevance to a site, such relevance having been established by aprecomputed site assessment based on aerial or satellite data, e.g.,such as one commonly finds on internet mapping services. Alternatively,the relevance has been provided by end user input when the system waspurchased or otherwise secured for service at the site. The flow of FIG.2 can then be used to generate an object catalog 211 for each camera tobe used at the site (step 508), the color camera being the default andtypically the most useful (a thermal camera can be used, but it may havea smaller set of object features with which to estimate object types andcharacteristics); this motivates the aforementioned a priori tour andsecurity settings, as, for example, a late-in-the-day installation ofthe system may not allow adequate sunlight for capturing a color camerapanorama and a thermal panorama is not wanted. Given an object catalogfor the site, the objects can be selected and prioritized forthermography (step 509); spaces are also then selected and prioritizedfor security 510. These priority settings are either provided throughdialogue with end users, e.g., a customer service phone call prior toinstallation after which time a customer service agent enters the datainto the user's profile for the site etc., or the priority settings arecomputed based on statistically derived risk factors that draw fromhistorical data from other users or other a priori data that can be usedin a decision tree or equivalent probabilistic framework.

The method proceeds by posting object catalog and priority lists to theremote server (step 511), which makes it possible to review data for theuser by a remotely located expert, e.g., at a remote computer 118, anycorrections for which could be approved and entered, after which timethey can be used to update the system automatically or manually. Thesystem then connects image streams to local and remote VMS (videomanagement system) resources (step 512). The local VMS analyzes andstores all relevant video data locally and transmits a subset of thedata to the remote VMS running on the remove server 115 such thatbandwidth is minimized on the network connection, e.g., LTE wireless inwhich case data is relatively expensive for users. The remote VMSenables a user to review historical/archival events and video segmentswhile also, as needed, viewing live video from a site, e.g., in theevent there is a security incident that requires observation. The localVMS is configured for recording and storing locally on a continuousbasis, up to some desired interval, e.g., 3 days, 1 week, 1 month, etc.

The system is now able to begin its duties (the sequence thus far havingtaken place in a matter of minutes, typically), beginning with executinga tour per its schedule while also enabling the use of calibrated data(step 513). This means that, in one embodiment, the gimbal will visiteach object cataloged in priority order and/or timing and collect data.Since the data collected is now thermographic, it must be calibrated inorder to be maximally useful. Thus the image streams are extended to orswitched to calibrated image streams. For each tour event sequence, thesystem will post thermography data to a remote server 115 (step 514) andthen post any auxiliary, internal, or external sensor data (from sensors105 and/or 113) (step 515) to the same server. The remote server 115 isthen able to update corresponding analysis and graphing features thatusers might access by way of a network connection and a user interface,e.g., browser display (step 516).

As one non-limiting example, FIG. 7 illustrates a thermal image of anenvironment associated with a site 700 corresponding to a public utilityin the U.S. during the day, and FIG. 8 is a plot of absolute temperaturedifference (° C.) for three phases of a portion of the objectspreviously identified in the image. In particular, FIG. 7 illustratessix objects (bushings, in this case) labeled A-F, which correspond tobounding rectangles 701-706, respectively. The objects being assessed inFIG. 8 (‘A’, ‘B’, ‘C’) are bushings for a primary operating at 345 KV,while objects ‘D’, ‘E’, and ‘F’ are the secondary phases. FIG. 8 thenvisualizes the temperature trends for the three phases (AB, BC, and AC)associated with the objects.

Referring again to FIGS. 1 and 5, as time progresses, the remote server115 makes assessments based on the new data sets and will send messages,alerts or alarms to users using a Network 107, e.g., local area network(LAN) or wide area network (WAN) such as the internet (step 517).

Upon completion of a thermography tour, the security function begins orcontinues, in which the system runs a security schedule and usesuncalibrated data (step 518) for its VMS related functions and services.Periodically, following the predetermined interval selected for thethermographic tour frequency, a tour will begin (step 519) and repeatthe steps of 507 to 512. In this way, the system produces autonomousthermographic and security services using a system embodiment such as isshown in FIG. 1 and using a flow such as is illustrated in FIG. 5.

Platform Supporting Autonomous Operation

In accordance with various embodiments, the autonomy of the system,especially for remote sites, is greatly assisted by the Platform 106features of the system, which are illustrated in the exemplaryembodiment illustrated in FIG. 6. Providing a wheeled platform that isreadily transported enables it to be used at multiple sites forrelatively short amount of times; this is helpful when short termsurveys are required to assess a site for marginal equipment or sitebehavior prior to a permanent installation, or if sites are somewhattransitory, e.g., in the case of mobile electrical substations.Likewise, the use of an extendable or telescoping mast or support forthe embedded system 100 enables greater ease of transport. Further,using renewable energy sources, e.g., wind or solar or equivalent,prevents the need for an electrical connection to a site, which in thecase of remote electrical substations, saves cost and time andregulatory burden. Additionally, using wireless networks having widearea coverage, such as is available through satellite or commercialtelecommunications services e.g., LTE, enables use of the system withoutthe need for a site network connection and therefore removes the needfor site network equipment.

With reference to the conceptual block diagram of FIG. 6, a platformthat supports autonomous operation 600 generally includes a supportframe with extensible supports 602 that together are used as afoundation and enable leveling and stabilization of equipment mounted onthe platform. This frame 602 is further supported by wheels or fixedtrailer supports 601 that enable either towing the platform or haulingit atop another trailer, whichever arrangement is desired by the users.The equipment integrated onto the support frame 602 includes anextendable mast 603 that supports the system 100 shown in FIG. 1, e.g.,the gimbal and cameras and electronics/optics they house or carry. Thesystem 100 is powered by way of a system power conditioning 609 unitthat obtains its power, in turn from a battery management, storageassembly 606 which maintains optimum battery voltages and manages therenewable energy source 605 generally. The renewable source is typicallysolar or wind, and can be replaced by a wired connection to traditional,e.g., 120-240V AC power source, or other, e.g., DC power source, ineither case representing an on-site source of power the precludes theneed for a renewable source. Since the platform and supported equipmentillustrated in FIG. 6 are often located at a remote geography away fromtraffic or physical security, it is helpful to have a local intrusiondetection system that continually monitors the platform (i.e., lookingdown from a mast mounted location or looking out from the base of theplatform, or instantiated as one or more unattended nearby groundsensors that are in communication with the system) for activity. Thislocal slew to cue sensor(s) and controller 608 uses the LANRouter/Switch 607 connection to command the cameras that are part of thesystem 100 to break away from their tour or security duties and collect,process and send imagery of the platform (step 604) to a remote serverand, if remote or local computers 115, 118 produce an intrusionassessment, send messages to users that are responsible for sitesecurity. This slew to cue capability, being a built-in function for thesystem 100 also enables site perimeter based cueing devices to be usedto cue the system over the system LAN or by way of an independentwireless connection, e.g., LTE or equivalent.

Automatic Configuration of a Site

In accordance with the invention, the site where the system is installedand used for safety and security monitoring can be mapped and assessedso that objects, e.g., equipment, and spaces within view of the systemare located and labeled, after which time they can be observed over timefor thermographic behavior and occupancy by humans or other movingobjects or organisms. The system is configured for the site in which itis located using capabilities illustrated in the embedded system 100 ofFIG. 1 and the sequence of operations illustrated in the flow chart ofFIG. 2.

With reference to FIG. 1 and FIG. 2, an embedded system 100 is used toassess the site of the system installation and produce a catalog ofobjects and spaces at the site to be monitored. The method of FIG. 2begins with the selection of a camera to use in panorama generation(step 200), e.g., a Thermal Infrared Camera 103 or a Color Camera 104 oran Auxiliary Sensor 111. The method continues by computing the camerafield of view (step 201), e.g., from known parameters such as focallength, pixel size, and number of pixels, or by retrieving it from amanufacturer database or other reference for the camera. This field ofview is then used to compute a gimbal step size (step 202), allowing forsome overlap between adjacent images, e.g., 10 percent of each of widthand height is often acceptable, so that subsequent image stitching ismore easily performed and to avoid losing data at image edges due togimbal mechanical tolerances. The entire volume of space can be surveyedthis way, e.g., typically a hemisphere, such that any objects viewablecan be catalogued and used for subsequent measurements and observation.The gimbal is then moved in accordance with the computed step sizes(step 203) and images collected at each step while the gimbal is paused.The images gathered in this gimbal step sequence are then used toconstruct a panorama (step 204) using, for example, a stitchingalgorithm., e.g., by matching features in images and registering oneimage with respect to another this way, that merges image data from twofields of view into a single field of view. This stitched panorama imagecan then be used to detect and classify objects (step 205) that arefound in the panorama using one of the many techniques available forobject detection, e.g., neural networks or model-based methods orcombinations of the two, for instance. This results in a list ofobjects, for which it is then possible to constitute object properties(step 206), e.g., object make or model, color, emissivity, etc. Theobject-system distances are then measured (step 207), using a rangingdevice or the system of FIG. 1 and the method of FIG. 3, so that eachobject in the site can be located accurately in three-dimensional space.The spaces, e.g., entries, exits, interiors, exteriors, ground, sky,etc., are then detected and classified (step 208). As it is sometimesadvantageous to have a human review the work of a computer, the objectsand spaces can then be reviewed with an operator (step 209) andsubsequently corrected (step 210) before adding the objects and spacesthus classified and localized to an observation catalog (step 211). Inso doing, various embodiments of the invention enable a site to beautomatically configured for monitoring with the system of FIG. 1.

In accordance with various embodiments of the present invention,computing system 101 (as well as any other functional modules describedherein) may implement or more machine learning models that undergosupervised, unsupervised, semi-supervised, or reinforcement learning andperform classification (e.g., binary or multiclass classification),regression, clustering, dimensionality reduction, and/or such tasksbased on the acquired images.

Examples of models that may be implemented by system 100 include,without limitation, artificial neural networks (ANN) (such as arecurrent neural networks (RNN) and convolutional neural network (CNN)),decision tree models (such as classification and regression trees(CART)), ensemble learning models (such as boosting, bootstrappedaggregation, gradient boosting machines, and random forests), Bayesiannetwork models (e.g., naive Bayes), principal component analysis (PCA),support vector machines (SVM), clustering models (such asK-nearest-neighbor, K-means, expectation maximization, hierarchicalclustering, etc.), linear discriminant analysis models.

Thermographic Measurements

In accordance with various embodiments of the invention, an intentionfor its uses is to produce estimates of physical temperature ofcatalogued objects, or parts thereof; this is the practice ofthermography. Thermography requires, at minimum, an assessment oftemperature over time. In keeping with the system of FIG. 1,thermography is practiced using a Thermal Infrared Camera 103. Such acamera produces images having pixel values that are proportional to theradiance of the objects in the field of view that correspond to thepixel values; imaged object radiance, for a thermal camera, e.g., oneoperating at wavelengths proximate to 10 microns, is proportional toimaged object physical temperature. Radiance is not identically equal tophysical temperature, however, as radiance captured with a thermalcamera is always only the apparent radiance, a physical observable thathas object physical temperature as one of several contributing factors.Other contributing factors include, for example, object thermalemissivity, atmospheric radiance, radiance of the imaging apparatus(i.e., lens, window, iris or stop, etc.), radiance of nearby orgeometrically related objects, wind speed, solar radiance and angle ofincidence (i.e., as implied by time of day and geographic location).Thus, in order to accurately estimate object physical temperatures (anobjective of the invention), it is necessary to remove from the thermalcamera image data the effects of contributing factors, which bydefinition do not correspond to object physical temperature.Traditionally, the practice of thermography with human operatorscomprises human measurement of thermal images and estimation, sometimessupported with separate measurements, of contributing factors, alongwith subsequent correction of thermal image data which yields anestimate of object physical temperature that has a known, or at leastintended, accuracy.

One goal of various embodiments of the invention is to automate thepractice of thermography. Consequently, the apparatus of FIG. 1 servesas a proxy for a human operator who is equipped with thermal imagingapparatus and an ensemble of other measurement devices, all of which aredeployed by the human operator to effect an accurate assessment ofphysical temperature and subsequent communication of the assessment tointerested parties for objects of interest (in this case, cataloguedobjects). The practice of thermography when well performed by a human isrepetitious, complicated, and requires significant physical,mathematical, and analytical skill and care (when poorly performed thepractice is sometimes referred to as “pencil whipping”). An advantage ofthe present invention, as a means and method of thermography automation,is that electro-mechanical systems under computer control are wellsuited to tasks that are repetitious and complicated.

Having thus described a system for acquiring measurements, an exemplarymethod for doing so will now be described. Generally speaking, given ameans of measuring apparent radiance of an object, R_(a), and given aknown relationship between physical temperature and radiance, e.g., sucha relationship being typically provided by a thermal camera manufacturer(those with greater accuracy usually referred to as “radiometric thermalcameras”), the digital image data produced by a thermal camera can bedescribed. For the sake of this discussion, it is assumed that thethermal camera comprises an assembly of digitizer, image sensor (e.g., afocal plane array), thermal lens, thermal window, in that order fromthermal camera focal plane toward the object. With that in mind anequation for the digital image data produced can be written as

$\begin{matrix}{R_{a} = {{\tau_{win}\left( {{\tau_{atm}\left\lbrack {{\epsilon \; R_{p}} + {\left( {1 - \epsilon} \right)R_{r}}} \right\rbrack} + {\left( {1 - \tau_{atm}} \right)R_{atm}}} \right)} + {r_{win}R_{rW}} + {\left( {1 - \tau_{win} - r_{win}} \right)R_{W}}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

Where the left-hand side of the equation is an object, apparentradiance, R_(a) (embodied as a digital image data comprising an ensembleof image pixels that correspond to objects in the field of view of thethermal camera), τ_(win) is the transmissivity of the thermal window,τ_(atm) is the transmissivity of the atmosphere between the object andthe thermal camera apparatus, ∈ is the object thermal emissivity, R_(p)is the physical or self-radiance of the object, R_(r) is the radiance ofthe object due to reflected energy, R_(atm) is the radiance of theatmosphere between the object and the thermal camera, τ_(win) is thereflectivity of the thermal window, if present, R_(rW) is the radianceincident on the thermal window that can be reflected, and R_(W) is theradiance of the thermal window itself (by virtue of its nonzeroabsorption and thickness). In practice, the quantities τ_(win), τ_(win)can be measured in the laboratory and used thereafter. The remainingvariables in the above equation for R_(a) are assessed using standardmodels (e.g., MODTRAN for the atmospheric contribution), assumptions, orusing instruments, e.g., a pressure/temperature/humidity sensor assemblyto make an assessment of atmospheric loss using an a priori formulation.With this in mind, an exemplary method can be delineated for calculatingobject physical temperature, e.g., enabled by posting thermography data(514 in FIG. 5), where the posted data comprise one or more image datasets and have pixel values D that have a known relationship to theapparent object radiance R_(a), e.g., given by

$\begin{matrix}{D = {\frac{A}{e^{B/R_{a}} - C} + E}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

Where A, C, E are constants determined in laboratory conditions (oftenby the thermal camera manufacturer); this is an instantiation of theso-called Planck equation for radiometry used to relate radiometricquantities to physical temperatures. Using Equation 2, it is thenpossible to measure D, use Equation 1, solve for and calculate R_(a) interms of D then proceed to use solve for and calculate R_(p) in terms ofR_(a), which is related to the physical temperature, e.g., in degreesCelsius or in Kelvin, by a proportionality constant provided by thecamera manufacturer (or obtained in a laboratory of one's own). Toproceed toward an algorithm for automated solutions, we simplifyEquation 1 by replacing known or static variables and variables forwhich we can insert an available physical observable, e.g., measured andmodeled atmosphere, window temperature, etc., we obtain

R _(a) =k ₁ [∈R _(p)+(1−∈)R _(r) ]+k ₂  Equation 3

The relation expressed by Equation 3 will serve to delineate anexemplary sequence of steps for automating the estimation of physicalobject temperature, as follows:

-   -   1. Capture thermal image data containing pixels corresponding to        the object of interest.    -   2. Register the image data to a predetermined reference image        for which object pixel locations are known.    -   3. Obtain measurement and model data needed to compute k₁ and        k₂.    -   4. Use predetermined object orientation data, e.g., from 3D        measurement data obtained in accordance with FIG. 3, prior 3D        mapping data, or surface orientation deduced from object        recognition and stored in the object catalogue with other object        identification data, etc., to establish surface orientation        (angle of orientation with respect to gravitational vertical and        horizontal) for the object pixels of interest    -   5. Use object orientation data to estimate angular dependencies        of emissivity, solar and background contributors, e.g., object        emissivity, solar radiance, periphery radiance, respectively,        using geometrical relationships well established and known by        those skilled in the art.    -   6. Obtain thermal image data for the hemisphere of measurement,        defined here as the thermal-camera-viewable region (that which        can be imaged with the field of view of the thermal camera and        the gimbal it is attached to, if present) extending from the        ground, e.g., beneath a thermal camera, to the sky overhead,        such that the radiance contributed by any point in in the        hemisphere within the viewable region can be estimated.    -   7. Alternately, obtain thermal image data for the hemisphere of        measurement at such points as can contribute, geometrically        (i.e., as implied by optical ray tracing of object pixels of        interest) to the object radiance.    -   8. Use hemisphere of measurement data, solar illumination data,        and other contributors to estimate R_(r), e.g., by regression or        other methods known to those skilled in the art.    -   9. Use known properties of the object surface, orientation (with        respect to the thermal camera) to estimate object emissivity, ϵ.    -   10. Use Equation 3 to calculate object radiance, R_(a).    -   11. Use object radiance to calculate the object temperature        using predetermined proportionality constants.    -   12. In this way the object radiance can be estimated to a first        order, and this approach can be used to produce automated object        temperature outputs, e.g., using a computer.

The above sequence of operations, or algorithm, can be extended furtherif all the objects observed in a physical setting are treated as asystem of radiators that may contribute to the observed radiance for anobject of interest. Given R_(ir), R_(ia), and R_(ip) as the i^(th)object reflected, apparent and physical radiance, and having measuredR_(ia) it is then possible to iterate for a solution producingapproximate R_(ir), R_(ia), R_(ip), where R_(ir) is the radiance of theobject of interest due to reflected energy, R_(ia) is the radianceobservable for the object—its apparent radiance, and R_(ip), is theobject internal or self-radiance—its own physical radiance. If ∈_(i) isthe emissivity of the i^(th) object, then the i^(th) object will haveapparent radiance R_(ia) such that

R _(ia)=∈_(i) R _(ip)+(1−∈_(i))R _(ir)  Equation 4

and since R_(ir) is

$\begin{matrix}{R_{ir} = {\sum\limits_{\underset{k \neq i}{k = 0}}^{n}R_{ka}}} & {{Equation}\mspace{14mu} 5} \\{then} & \; \\{R_{ia} = {{\epsilon_{i}R_{ip}} + {\left( {1 - \epsilon_{i}} \right){\sum\limits_{\underset{k \neq i}{k = 0}}^{n}R_{ka}}}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

where R_(ka) is the k^(th) object radiance that can illuminate thei^(th) object and reflect, producing a contribution to R_(ia), thei^(th) object apparent radiance. In these calculations, it is implicitthat mathematical integration over solid angles occurs when necessary toproduce irradiances from radiances, and that integration over physicalareas is implied when producing power from irradiance. These integralsare not discussed explicitly here for the sake of simplicity. In all ofthese mathematical expressions, it is understood that the variables,e.g., R_(ka), can be represented by scalar or vector quantities,including treating such radiances as point-wise time series, twodimensional arrays (images), or three dimensional collections of twodimensional images e.g., stacked temporally as a “Datacube”. Further,given such a Datacube, time series of individual pixels (or groups ofpixels, or both) can be extracted from it and used to estimate trends inthe underlying physical temperature behaviors for the object implied bythe chosen pixels. Thus, whether the radiance variables ae scalars,spatial vectors or time vectors, these equations can be used toconstruct iterators, regressions and systems of equations that supportsolutions for R_(ip), using measurements that support R_(ir) and R_(ia).In such iterations or regressions, it is helpful to have multipledistinct data situations to provide insight and enable solutions formultiple variables. Some pairs of situations or variable situationscontemplated by this invention are:

-   -   Day vs. night, Ambient hot vs cold, Energized vs. not energized,        Wind vs. no wind, Background hot vs. cold, Shade vs. no shade,        Vertical vs. horizontal emitter surface (or, simply not        vertical), Load current high vs low, conductor thickness or        conductivity, Multiple points at same surface (i.e.,        uniformity), Ground data, Sky data, Material class or type.

In all of the above, it is recognized that solving for R_(ip) and/or∈_(i) can be improved through the use of known/good physics, or physicalmodels of underlying phenomena. In such cases, physical models areincorporated into the framework of Equation 1-6 and solutions, e.g.,regression for coefficients or parameters, can be produceddeterministically and/or stochastically as befits the particularsolution scenario. It is also recognized that solution structures willsometimes benefit from neural networks including or modeled aftersystems of equations, e.g., Equation 4-6. The mapping of such equationsto neural networks, e.g., those deterministic and/or stochastic innature, and the methods of solution are well known to those skilled inthe art.

Image Registration

In accordance with the invention, the objects at the site where thesystem is located will be imaged recurrently so as to assess theirphysical temperature over time. In order to assure consistency ofmeasurement and given that the field of view of the camera(s) used mayvary with time, e.g., owing to gimbal movement repeatability or due tomotion of the support structure for the camera(s), the use of imageregistration, or pixel mapping, is contemplated with the invention. Inthe registering of images, a suitable reference image is selected for aninterval of time, e.g., manually or using predetermined imagecharacteristics to automatically select an image, and then all otherimages in that period of time are spatially adjusted so as to collocateobjects by pixel location in space and time.

There are numerous methods for automatically registering images that arewell known to those skilled in the art. In order to avoid the need forhuman intervention to produce registration results using such well knowntechniques, it is important to automatically identify image features,particularly locations where objects of interest are stationary and canreliably be used for registering temporally distinct images. A preferredembodiment for finding stationary objects uses a Datacube of thermalimagery, representing a time series of two-dimensional thermal images,as a means of identifying features to register, as follows:

-   -   1. Compute an edge representation of the data cube using spatial        filtering of each 2D image contained in the Datacube, e.g., a        Sobel or similar two-dimensional derivative based technique.    -   2. Binarize the edge representation of the Datacube.    -   3. Temporally integrate the Datacube for multiple temporal        statistics, e.g., minimum, maximum, mean and variance, and use        these to assess the spatio-temporal stability of edge features.    -   4. Identify edge features having low spatio-temporal variability        and high levels of occurrence, such features representing the        most probable stationary features.    -   5. Iterate through the edge features, selecting those with most        favorable statistics first, masking these features from further        consideration, and progressing to the next most favorable        feature, etc., until the feature list is exhausted.

Having automatically produced a reliable set of image features, one ofmany image registration algorithms can be used to map edge features in agiven image to corresponding image features in a reference image, themap thus produced permitting the calculation of corrections to apply tothe given image to enforce spatial correspondence to the referenceimage.

While the preferred embodiment of feature selection makes use of edgefeatures to identify features, it is here contemplated that othermorphological features, e.g., corners, rectangles, circles, othernon-geometric features having measurable statistics, can be used in asimilar fashion to produce sets of features that can be used to comparepairs of images and spatially register one to another.

Further, given robust and accurate image registration techniques, it iscontemplated that the invention will be used to locate objects withsub-pixel accuracy over time, which, in turn, enables the assessment ofpixel-scale temperatures that would otherwise be impossible to observereliably over time.

Corona Mapping

In accordance with one embodiment of the invention, object imagery isproduced over a time period for diverse objects of interest, includingelectrically energized objects having electric field intensitysufficient to ionize air molecules (corona discharge) in proximity tothe invention, e.g., in the air space near high voltage transformerconductors or bushings. This ionization is observable with the inventionas collections of point sources that appear as a cloud-like structure inthermal imagery. The spatio-temporal behavior of the ionization is anindicator of state for the energized apparatus and can be used to makeassessments of apparatus state that support predictive maintenance andfailure onset. The invention thus contemplates the use of thespatio-temporal behavior of imagery of ionization events, e.g.,performed using existing morphological detection and tracking algorithmsknown to those skilled in the arts of computer and machine vision, toassess the physical condition of proximate energized structures anddevices.

Image Temporal Evolution

In accordance with various embodiments of the invention, object imageryis produced over a span of time such that temporal effects in 2D and 3D,e.g., when range data is integrated with thermographic data, can beobserved. This corresponds approximately to the use of time lapse video,which when applied to thermographic data, can include analyzing timelapse video as a class of object detection and tracking, the object inthis case being a region of temperature change, e.g., hot spot or coldspot, that can propagate in an electrical circuit and the structuresassociated with it. Treating three-dimensional heat propagation as aThermal Object detection and tracking problem enables the re-applicationof many robust and mature algorithms in the domains of machine andcomputer vision. The invention contemplates the fact that differentobject classes, e.g., switches, fuses, arrestors, bushings, will havedistinct shapes of heat propagation and that, as these shapes evolveover time, they will constitute different “motions” for the thermalenergy that is propagating. As such, algorithms presently in use totrack and interpret human behavior based on motion sequences can beapplied to event detection in a thermographic setting. For instance, inthe same way that patterns of human hand motion can be interpreted asvarious signals, e.g., sign language or commonly recognized gestures,patterns of heat propagation can be interpreted as various physicalphenomena, e.g., loose connector, cracked bushing, motor bearingfatigue, etc.

Further, again viewing temporal sequences of thermal image data for ascene as a 3D Datacube, calculating a 2D image of pixel-wise temporalbehaviors, e.g., mean, variance, intra-scene correlation, frequencydomain filters, or other metrics derived from comparisons to physicalmodels, allows identification of object features of interest in thespatial domain—temporal behavior can be detected using the spatialdomain. Features of interest in this type of analysis includenon-energized surfaces, surface emissivity and air convection surfaces.Such a view of data also permits quick analysis of trends betweenobjects, such as the temperature differences between bushings for thethree phases of a distribution transformer.

Deduction and Use of Site Schematic Data

In accordance with the invention, the objects at the site image data aregathered are often related to one another as elements of an electricalcircuit. When this is the case, one can use the invention to capture 3Dinformation about the viewable objects and support structures toconstruct a circuit diagram; alternately, a circuit diagram can beaccessed from separate 3D observations or site design data. Given such acircuit diagram, the objects viewed and identified, e.g., manually orwith computer vision techniques, at the site can be associated withcircuit features. The thermal data subsequently gathered for objects canbe used to interpret electrical loads using known physics, e.g., Ohm'sLaw, nodal analysis, and other analytical tools known to those skilledin the art of circuit behavioral analysis. Such treatment of the sitedata also enables the use of thermal data to support so-called “digitaltwin” strategies, wherein sensor data gathered for a designed system areused to update companion physical models of the system such that systemstate in the present and future can be estimated and exploited, e.g.,for the assessments of state root causes or collateral effects.

Incorporation of Collaborative Sensors

In accordance with the invention, given connectivity permitted by acommunications network or the signal connections of the computerprocessor that is integral to the invention, a multiplicity of sensorscan be used to make assessments of site state as a function of time. Forexample, video security systems or unattended ground sensors (UGS) inproximity to the installed invention can be used to cue the inventionfor monitoring intrusions at the site. Alternately, UGS havingcalibrated thermal sensors, e.g., spot sensors, integrated into thestructure that supports the invention or located independently and inproximity to the invention, can be used to either cue the invention tothe presence of intrusion or thermal events or can be used for groundtruth that supports algorithmic techniques for constraining solutions,e.g., for object physical temperature or emissivity. Further, asinstallation sites may often have other independent data collectionsystems the data from these may also be used by or with the invention tofocus the observations on regions of heightened interest, e.g., hotspots or locations of probable anomalies.

Use of Reference Points as Constraints

In accordance with the invention the observed behavior of ThermalObjects, including with reference to independent measurements, e.g.,spot measurements with hand instruments or additional devices integratedwith the invention, e.g., UGS, will produce assessments of site regionsfor which there is elevated accuracy and reliability. By integratingindependently collected comparison data for objects or deducing thesefrom temporal behavior, e.g., permitted by Datacube analysis, it iscontemplated that anchor points for constraining solutions for R_(ip)and/or ∈_(i), for instance, can be automatically produced. Using suchhigh confidence points enables more robust solutions by addingnon-spurious information to the solution spaces. And in simple cases, itenables the automation of inter-object relative thermal trending.

Use of Scene Based Optical Characterization

In accordance with various embodiments of the invention, the objectsthat can be observed include, without limitation, the sun, moon, starsand other known point sources. In order to optimize the resolution ofthe system, it is contemplated that known point sources can be used toestimate the optical performance over time as concerns resolution, e.g.,the point spread function (PSF) or equivalently the modulation transferfunction (MTF). Knowing such behaviors permits improving the resolutionand thereby the thermal accuracy of the system using techniques known tothose skilled in the art, e.g., deconvolution or more sophisticatedtechniques such as the CLEAN algorithm, etc. Further, by observing theoptical behavior over time, after accounting for known atmosphericvariables, e.g., water vapor content, the invention can be used todeduce the optical effects of actual atmosphere conditions along theoptical path, e.g., the blur induced by multiple scatter in the verticalatmosphere vs. the horizontal atmosphere. Finally, knowing opticalparameters for the invention and its environment supports improved imageoptimization such as super-resolution.

Object Distance Measurements

In accordance with various embodiments of the invention, the objects atthe site where the system is located may be assessed for their distancefrom the invention, such that objects can be accurately placed inthree-dimensional space, e.g., global position data, in order to makefurther measurements of physical properties, sizes and relationships ofobjects by themselves and in relation to other objects.

The system is configured for the site in which it is located usingcapabilities illustrated in the embedded system 100 of FIG. 1 and thesequence of operations illustrated in the flow of FIG. 3 is used tocompute object-system distances. This sequence is executed so as togenerate a sequence of stereo-pair images, it being known in advancethat stereo-pair imagery can be used to deduce distance relationshipsbetween an imaging apparatus and an object if the object coordinates(corresponding to physical distances in the focal plane of the camerabeing used) in the images and the physical image device focal planeseparation are known. The relationship between these two distances isdescribed as a disparity function and is well known by those skilled inthe art. In the simplest case it is described by z=f*b/d, where z is thedistance to the object, f is the focal length of the camera optics, b isthe separation between images (focal planes) and d is the distancebetween objects in the stereo-pair images. In the present case, a singlefocal plane is placed at multiple physical separations by virtue ofgimbal motion. This motion is usually rotary but is still effectivesince rotary motion produces translation in proportion to the radius ofcurvature of the arc of motion and the angular extent of the arc ofmotion. Because the motion is rotary however, the relationship z=f*b/dis approximate and will have additional nonlinearities to address as aresult of using rotation to produce displacement.

With reference to FIG. 1 and FIG. 3, an embedded system 100 is used toactuate (move) a gimbal in one or more axes of motion to discretelocations and capture one or more images at each position, such that astatistically significant variation in image feature locations (e.g.,two dimensional pixel coordinates) is observed as a function of gimbalposition (e.g., angular position of one or more axes) in order that aregression may be performed to deduce the relationship between featurelocation and gimbal position. This regression can be used with adisparity function to compute object range from the slope of theregression—the slope of the regression (which is b/d in the relationshipz=f*b/d) being proportional to the object range.

The method illustrated in FIG. 3 proceeds by centering the camera fieldof view on the object for which distance is to be measured (step 300),capturing an initial image (step 301) and locating the object features302 in that image. This first image will be a reference to whichsubsequent images will be compared, and differences computed, as thegimbal is moved away from its initial position. The flow proceeds bythen selecting a gimbal axis of motion and related position increment(step 303) and executing a sequence repeatedly: capture image (step305), locate object features in the image (step 306), store featurelocations (pixel coordinates) and corresponding gimbal position (step307) (gimbal angular coordinates), and then attempt a regression and asignal to noise ratio (SNR) (step 308), where SNR here is a statisticformed by the ratio of the major axis of the ellipse formed by thetwo-variable regression data scatter (as one encounters in atwo-variable scatter plot) to the minor axis of the ellipse. In thiscase we are considering “signal” to be the object feature displacementand “noise” is the scatter of signal perpendicular to the regressionline drawn through the plot of object feature displacement vs. gimbaldisplacement. This example is assuming a perfectly linear relationshipfor simplicity of discussion; it is contemplated that the relationshipwill be nonlinear. The steps 304 through 308 are then iterated until theSNR is larger than a predetermined threshold (step 309). If anadditional axis of motion is to be used, this decision 310 can beexecuted and the sequence of 303 through 310 can be repeated until allaxes of motion have been explored, including axes that are combinationsof principal gimbal axes of motion. The flow is completed when gimbalaxes have been exercised to produce displacement regressions for objectsof interest, after which time the regression outputs are stored (step311) and object distances are computed (step 312) from the relevantdisparity function for the gimbal and camera.

There are many ways to improve the distance estimates. One that iscontemplated for this system 100 including within the flow of FIG. 3,e.g., capture image (step 305), is the use of super-resolutiontechniques (e.g., using the flow of FIG. 4) to improve the resolution ofthe object displacement in image coordinates, the super-resolution beinga means of computing new, smaller equivalent pixels in a focal planeusing sub-pixel angular movements of the gimbal and, as needed,de-blurring of images based on known or measured optical properties ofthe camera lens (e.g., lens point spread function). In this way, thedistance measurement may be improved.

Another way to improve distance estimates is through the use of imageaveraging, or stacking, as it is sometimes known, to increase the signalto noise ratio in an image by effectively increasing the integrationtime for each pixel in the image. As with the use of super-resolution,averaging finds use in the flow of FIG. 3 during capture image 305.

Image Super Resolution

In accordance with the invention, the objects and spaces observed withthe invention, may be observed with greater fidelity, either forthermographic or intrusion purposes, with increased image resolution,e.g., more pixels per image or more pixels per degree of optical viewingangle. A known technique for achieving this purpose is super-resolution.Generally speaking, this technique involves combining multiple images ofa scene, collected at different viewing angles, and subsequentlycombining these images so as to improve the resolution of the originalimage, effectively computing additional image pixels containing newinformation, that information being provided by other images.

The system is configured for the site in which it is located usingcapabilities illustrated in the embedded system 100 of FIG. 1 and thesequence of operations illustrated in the flow of FIG. 4 is used tocompute new pixels for an original image, such that a newsuper-resolution image is produced. The flow of FIG. 4 proceeds by firstcentering the gimbal on an area of interest (step 401), specifying theresolution desired as a multiple (N) of the unimproved resolution of thecamera (step 402), and then computing a gimbal increment (in gimbalposition coordinates) that will produce the desired resolution multiple(step 403) and finally capturing an image that is at the center (step404) of the to-be-generated super-resolution image.

A decision can be made at this point in the flow as to whether tocollect the maximum number of images required for super-resolution, orN{circumflex over ( )}2 (=N×N) or whether a lesser number of images isto be used (step 405). The advantage to using fewer images is the speedand complexity of motions and subsequent super-resolution mathematics;the advantage to using all the images is that available super-resolutionimage data will be maximized. If N{circumflex over ( )}2 images isselected, then the gimbal is programmed for all N{circumflex over ( )}2positions (step 406), images are collected at each position (step 407),effects of the known point spread function (PSF) of the camera aredeconvolved (step 408) using one of many available techniques for PSFbased image improvement, after which operation the flow computes thesuper resolved image from the image set (step 409) thus obtained. In asimilar fashion, if all images are not to be used (step 405) then amovement pattern of less than N{circumflex over ( )}2 is programmed intothe gimbal (step 410), images are collected at each of these positions(step 411), the PSF is again deconvolved from the image data (step 412),images not collected are interpolated from the available images (step413) and a super resolved image is computed (step 414) from the imagescollected and PSF-corrected.

In accordance with various embodiments, super-resolution such as theexample flow of FIG. 4 can also be supported or replaced usingadditional cameras. For instance, images from a camera with oneresolution and another camera with double the resolution but half thefield of view could be combined into a single, higher resolution imageof the same field of view as the lower resolution camera, such that thehighest resolution occurs where the fields of view overlap, e.g., nearthe center of the image, the non-overlapping regions of the resultantimage having its increased resolution derived from interpolating theoriginal lower resolution image. In such instances, the camera PSF canbe used to deconvolve imagery prior to combining images so as tomaximize the insertion of new information into the resultant higherresolution image.

Additionally, if the camera with lower resolution and wider field ofview is radiometrically calibrated, a higher resolution calibrated imagemay be computed by combining a relatively uncalibrated high-resolutionimage with the calibrated low resolution image, using the low resolutionimage as a “tie point” for the calibration of each pixel. In this way acostlier calibrated device may be used to produce enhanced imagerywithout the expense of a larger focal plane array.

Furthermore, if a panorama or wide area image is formed with a highresolution, narrow field of view sensor, e.g., one that may be low costand relatively uncalibrated, this panorama may be used in combinationwith a lower resolution camera to produce higher resolution images atthe time of subsequent measurements, by interpolating new, higherresolution pixels for combination with/into the lower resolution and,typically, calibrated image. This approach assumes a relatively staticbackground condition for the panorama, e.g., a space or equipmentassembly that does not move with time, so that the shape of the spacerepresented in the panorama can be used to produce a similar shape inthe otherwise unresolved pixels of a lower resolution image that occurswithin the image extend of the high resolution panorama. This techniquebenefits from knowing the mapping of the instantaneous field of view foreach pixel of each of the cameras (e.g., low resolution and highresolution) with respect to one another, and their PSF as a function oflocation in the focal plane array, such as can be obtained through aboresight alignment and optical characterization laboratory measurement.

In summary, an automated thermal imaging system in accordance with oneembodiment includes: a thermal infrared camera configured to producethermal images of objects at a site within its field of view; a gimbalassembly coupled to the thermal infrared camera, the gimbal assemblyconfigured to move the thermal infrared camera to thereby adjust thefield of view of the thermal infrared camera; a network interface; apower source; and a computer processor communicatively coupled to thethermal infrared camera, the gimbal assembly, the network interface, andthe power source. The computer processor is configured to send positioninstructions to the gimbal assembly, capture a plurality of thermalimages from the thermal infrared camera, produce a panorama image of thesite based on the plurality of thermal images, detect and classify a setof objects of interest within the panorama image, produce state datacharacterizing the temperatures of the objects of interest, and transmitthe state data to a remote server via the network interface.

In accordance with one embodiment, the system further includes a mobileplatform configured to allow repositioning of the automated thermalimaging system to a selected site.

In accordance with one embodiment, the computer processor is configuredto perform a self-configuration procedure based on objects detected andclassified at the site during set-up, substantially without humanintervention.

In accordance with one embodiment, the power source is a renewableautonomous power source drawn from the environment at the site.

In accordance with one embodiment, the computer processor is configuredto perform the detection and classification of objects of interest usingat least one machine learning model.

In accordance with one embodiment, the computer processor is furtherconfigured to perform intrusion detection based on the plurality ofthermal images and send an alarm via the network interface when such anintrusion is detected.

In accordance with one embodiment, the system further includes at leastone auxiliary GPS sensor configured to sense the location of the thermalimaging system and utilize that location data in producing the statedata.

In accordance with one embodiment, the computer processor is furtherconfigured to use a Datacube time-series data structure for determiningthe state data.

In accordance with one embodiment, the computer processor is furtherconfigured to estimate corona effects for a high-voltage object ofinterest.

In accordance with one embodiment, the computer processor is furtherconfigured to perform a resolution enhancing process on the acquiredthermal images.

Embodiments of the present disclosure may be described herein in termsof functional and/or logical block components and various processingsteps. It should be appreciated that such block components may berealized by any number of hardware, software, and/or firmware componentsconfigured to perform the specified functions. For example, anembodiment of the present disclosure may employ various integratedcircuit components, e.g., memory elements, digital signal processingelements, logic elements, look-up tables, or the like, which may carryout a variety of functions under the control of one or moremicroprocessors or other control devices.

In addition, those skilled in the art will appreciate that embodimentsof the present disclosure may be practiced in conjunction with anynumber of systems, and that the systems described herein are merelyexemplary embodiments of the present disclosure. Further, the connectinglines shown in the various figures contained herein are intended torepresent example functional relationships and/or physical couplingsbetween the various elements. It should be noted that many alternativeor additional functional relationships or physical connections may bepresent in an embodiment of the present disclosure.

As used herein, the terms “module” or “controller” refer to anyhardware, software, firmware, electronic control component, processinglogic, and/or processor device, individually or in any combination,including without limitation: application specific integrated circuits(ASICs), field-programmable gate-arrays (FPGAs), dedicated neuralnetwork devices (e.g., Google Tensor Processing Units), electroniccircuits, processors (shared, dedicated, or group) configured to executeone or more software or firmware programs, a combinational logiccircuit, and/or other suitable components that provide the describedfunctionality.

As used herein, the word “exemplary” means “serving as an example,instance, or illustration.” Any implementation described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other implementations, nor is it intended to beconstrued as a model that must be literally duplicated.

While several illustrative embodiments of the invention have been shownand described, numerous variations and alternate embodiments will occurto those skilled in the art. Such variations and alternate embodimentsare contemplated and can be made without departing from the spirit andscope of the invention as defined in the appended claims.

1. An automated thermal imaging system comprising: a thermal infraredcamera configured to produce thermal images of objects at a site withinits field of view; a gimbal assembly coupled to the thermal infraredcamera, the gimbal assembly configured to move the thermal infraredcamera to thereby adjust the field of view of the thermal infraredcamera; a network interface; a power source; and a computer processorcommunicatively coupled to the thermal infrared camera, the gimbalassembly, the network interface, and the power source; wherein thecomputer processor is configured to send position instructions to thegimbal assembly, capture a plurality of thermal images from the thermalinfrared camera, produce state data characterizing the temperatures ofthe objects of interest, and transmit the state data to a remote servervia the network interface.
 2. The automated thermal imaging system ofclaim 1, wherein the computer processor is further configured to producea panorama image of the site based on the plurality of thermal imagesand detect and classify a set of objects of interest within the panoramaimage.
 3. The automated thermal imaging system of claim 1, furtherincluding a mobile platform configured to allow repositioning of theautomated thermal imaging system to a selected site.
 4. The automatedthermal imaging system of claim 1, wherein the computer processor isconfigured to perform a self-configuration procedure based on objectsdetected and classified at the site during set-up, substantially withouthuman intervention.
 5. The automated thermal imaging system of claim 1,wherein the power source is a renewable autonomous power source drawnfrom the environment at the site.
 6. The automated thermal imagingsystem of claim 1, wherein the computer processor is configured toperform the detection and classification of objects of interest using atleast one machine learning model.
 7. The automated thermal imagingsystem of claim 1, wherein the computer processor is further configuredto perform intrusion detection based on the plurality of thermal imagesand send an alarm via the network interface when such an intrusion isdetected.
 8. The automated thermal imaging system of claim 1, furtherincluding at least one auxiliary GPS sensor configured to sense thelocation of the thermal imaging system and utilize that location data inproducing the state data.
 9. The automated thermal imaging system ofclaim 1, wherein the computer processor is further configured to use aDatacube time-series data structure for determining the state data. 10.The automated thermal imaging system of claim 1, wherein the computerprocessor is further configured to estimate corona effects for ahigh-voltage object of interest.
 11. The automated thermal imagingsystem of claim 1, wherein the computer processor is further configuredto perform a resolution enhancing process on the acquired thermalimages.
 12. A method for automated thermal imaging, the methodcomprising: providing a thermal infrared camera configured to producethermal images of objects at a site within its field of view; securing agimbal assembly coupled to the thermal infrared camera, the gimbalassembly configured to move the thermal infrared camera to therebyadjust the field of view of the thermal infrared camera; providing anetwork interface; providing a power source; and providing a computerprocessor communicatively coupled to the thermal infrared camera, thegimbal assembly, the network interface, and the power source; sendingposition instructions from the computer processor to the gimbalassembly; capturing a plurality of thermal images from the thermalinfrared camera; producing a panorama image of the site based on theplurality of thermal images; detecting and classifying a set of objectsof interest within the panorama image; producing state datacharacterizing the temperatures of the objects of interest; andtransmitting the state data to a remote server via the networkinterface.
 13. The method of claim 12, further including providing amobile platform configured to allow repositioning of the automatedthermal imaging system to a selected site.
 14. The method of claim 12,further including performing a self-configuration procedure based onobjects detected and classified at the site during set-up, substantiallywithout human intervention.
 15. The method of claim 12, wherein thepower source is a renewable autonomous power source drawn from theenvironment at the site.
 16. The method of claim 12, wherein thedetection and classification of objects of interest is performed usingat least one machine learning model.
 17. The method of claim 12, furtherincluding performing intrusion detection based on the plurality ofthermal images and sending an alarm via the network interface when suchan intrusion is detected.
 18. The method of claim 12, further includingusing an auxiliary GPS sensor to sense the location of the thermalimaging system and utilize that location data in producing the statedata.
 19. The method of claim 12, wherein the computer processor uses adatacube time-series data structure for determining the state data. 20.An automated thermal imaging system comprising: a thermal infraredcamera configured to produce thermal images of objects at a site withinits field of view; a gimbal assembly coupled to the thermal infraredcamera, the gimbal assembly configured to move the thermal infraredcamera to thereby adjust the field of view of the thermal infraredcamera; a network interface; a power source; a mobile platformconfigured to allow repositioning of the automated thermal imagingsystem to a selected site; a computer processor communicatively coupledto the thermal infrared camera, the gimbal assembly, the networkinterface, and the power source; wherein the computer processor isconfigured to: send position instructions to the gimbal assembly;capture a plurality of thermal images from the thermal infrared camera;produce a panorama image of the site based on the plurality of thermalimages; perform a self-configuration procedure based on objects detectedand classified at the site during set-up, substantially without humanintervention; detect and classify, using one or more machine learningalgorithms trained during a set-up operation, a set of objects ofinterest within the panorama image; produce state data characterizingthe temperatures of the objects of interest; and transmit the state datato a remote server via the network interface.